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Forney's asymptotic upper bound for per-bit error probability in the 
detection of pulse-amplitude-modulated digital data in the presence of 
additive white qaussian noise was obtained for the case where the dura- 
tion of the intersymbol interference is bounded. In this paper, we show the 
validity of Forney's bound under much weaker assumptions that allow 
unbounded intersymbol interference. 

I. INTRODUCTION 

We consider the situation where a data sequence cto, • • ■ , aN-i of ±l's 
is transmitted via pulse amplitude modulation as Y,k-o h(t — kT)a k 
and received in the presence of additive white gaussian noise with one- 
sided spectral density a 1 . In a recent series of papers, Forney, 1 
Foschini, 2 and Mazo 3 developed an asymptotic (as o- 2 — ► 0) upper 
bound on the error probability per data bit P e : 

P e ^expj-^[l + (l)]j, (1) 

where d(h) is the minimum £■> distance between distinct modulated 
pulse sequences. This bound holds under the strong assumption that 
the pulse h(t) is supported on finite interval. 

In this paper, we show that (1) is valid for a considerably wider class 
of h(t). Roughly speaking, our assumptions are little more than that 
h(t) is in £i(- », «) and JC 8 (— «>, *>), and that H(f), the Fourier 
transform of h(t), does not vanish on an interval. The precise conditions 
on h(t) under which (1) holds are given below. In particular, (1) is 
valid when H(f) is a rational function. 

In Section II we give a precise statement of our results, and the proof 
follows in Section III. 

II. FORMAL STATEMENT OF PROBLEM AND RESULTS 

In this section, we give a precise statement of the problem and the 
results that were stated informally in Section I. 

We begin with some definitions. We denote -V vectors by boldface 
superscripted letters, and components by subscripted letters, e.g. 
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u -v = ( W0) . • . f w A r_i). When the dimension N is clear from the context, 
we omit the superscript. Define the sets (In, Gjv.*, Q-N.k by 

Q.S = {u' v : uj = ± 1, ^ j ^ N - 1|, 
at, k = {u' v : u k = + 1, ifc - ± 1, j * A|, (2) 

a*.* = {u' v : M fc = - 1, Mi = =fc 1, j 3* *}. 

Of course, a N = ttfe U Clj7,*- Again, when N is clear from the context, 
we write (In = Cfc, a^, fc = &£, G.N, k = ®jT- 

Next, let f(t), g(t), and — «> < t < oo be real- valued measurable 
functions. The inner product of / and <y is denoted by 

</, .7) - £ /(Of CO*, (3a) 

and the norm of / is 

ii/ii = «/,/»* -(£/■«**)* (3b) 

For a vector u v E Ctjv, and /(f), — °° < J < °°. a real-valued function, 
let the function h*u = s be defined by 

s(0 = N £ fit - *T)«*. 

where T > is a fixed parameter. 

We are concerned with the following modulation scheme. Let 
a n = ( ao> . . lf aAr _j) E Gat denote the data to be transmitted. Assume 
that all the 2 N vectors in a N are equally likely. The transmitted signal 
is the function h*a. N , where the pulse h(t) is a fixed function for which 
||/t|| < oo. The received signal is 

y(t) = (h*SL N )(t) +z{t), -cc<f<co, (4) 

where z(t) is a sample from a white gaussian noise process with zero 
mean and one-sided spectral density a 2 . 

The decoder associates with the received signal y, a vector 
B(y) = k N E Gat. Corresponding to a given decoder function D, let 
the bit error probability be 

P eN (D) =^ J E 1 Pr{d i ^a,}. (5) 

iv k=o 

Also, define the optimum error probability 

P*eN= PUh, o*> = miP eN (D). (6) 

D 

We are concerned here with the asymptotics of P* eN (h, <r 2 ), as cr 2 — * 0, 
i.e., as the signal-to-noise ratio approaches infinity. Accordingly, define 

E N (h) = - lim inf a 2 log P* N (h, a 2 ), (7a) 
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so that, as a 2 — > 0, 

E N (h) 



Pex(h, <r 2 ) ^ exp 



J 



[1 + 0(1)] ■ (7b) 



Next, consider a particular decoder that is of special interest here — 
the maximum-likelihood decoder, denoted Z) A . In the present problem, 
Dh{y) can be taken to be that u £ &n such that for all u £ CL N , 
u ^ u, 

IIj/i - //*u||<|!?/i - /j.*u||, (8) 

where yi is the projection of y onto the subspace of £>2(— °°, °°) 
spanned by the signals /;*u, u £ (Jat. With probability 1, (8) will be 
satisfied for some u £ ft^- 

Now, subject to the condition that h(t) has finite support, i.e., there 
exists a to > such that 

h(t) = 0, for |2[ > U, (9) 

Forney, 1 Foschini, 2 and Mazo 3 have shown that E(h) ^ d?(h)/4, where 
the "minimum distance" d(h) is defined by 

d(h) = lim inf min \\h*u — h*v\\. (10) 

N—«> u.veGiv 

U?£V 



Thus, as a 2 — * 0, 

(11) 



P*eAh, a") ^ exp I- *ffi [l + o(l)] 

Inequality (11) is established by showing that the error probability 
for the maximum likelihood decoder, P e jv(D/,), is overbounded by the 
right member of (11). This is done by writing (this is not as difficult as 
it looks) 

PMDk) = i Z E 2-<*-» Pr {£»(?/) 6 aj7.*|y = h*u + 2} 

4l I 2-'-v-D Pr { u |D»(y)-vJ|y-A«ii + «) 

^ v * uea» + Tea* 

Pr| U [\\yi-h*a\\^\\ yi -h*v\\}\y = h*a + z\ 

= ^Z L 2-<^-»Pr U [<*, Hv-u)) 

^i||Mv-u)|i) 
= *jr(A, <x 2 ). (12) 

Relation (12) is valid for any h(t). Subject to condition (9), it is then 

UNBOUNDED INTERSYMBOL INTERFERENCE 1343 



shown that, as a 2 —* 0, 



Mi, v 2 ) ^ exp 



^[1-WD] 



[13) 



where oi(l) does not depend on N. Thus, since P* eN ^ P e N(D h ), (11) 
holds. Further, the o(l) term in (11) does not depend on A'. An 
interesting by-product of these results is that the performance indi- 
cated in (11) is achievable via the decoder D h . This decoder can be 
instrumented (using the Viterbi algorithm) with a complexity which 
remains bounded as N — * <*> . 

We now drop the assumption that h(t) has finite support. Instead, 
we assume that h (t) satisfies the following conditions : 

(i) There exists a nonnegative £i function g (t), i.e., 



F. 



(, (t)dt <co, 

such that 

\h(t)\ ^ (jo(t), -*< t <cc, (14) 

and such that go is monotone in \t\. 

(ii) Let 

H(f) = r h(t)e- i2 *"dt, - oo < / < oo (15) 

be the Fourier transform of h(t). By (t), / 1 h(t) \dt < oo , so that H(f) 
is well defined for all /. We assume that there exists a nonnegative £i 
function G x (f) which is monotone in | /| , such that 

\H(f)\ z SG 1 (f), -co</<«o. (16) 

(Hi) Let the "folded spectrum" of h be 

su)= i k(/+S)r. og/ii. a?) 

n= — » I \ -* / I 

We show in Appendix A that S(f), ^ / ^ 1/7 7 , is finite and con- 
tinuous. We assume that 8(f) > 0, g / g 1/T. Let 

m = min 5(/) > 0, (18) 

where the existence of the minimum follows from the continuity of 
8(f) on the compact interval [0, 1/T]. 

Remarks: 

(1) Condition (i) is just slightly stronger than simply requiring h 
tobein£i(- oo, oo). Condition (14) forces h(t) to go to zero as |/| — »■ oo 
in a "well-behaved" manner. Condition (ii) imposes a similar condition 
on|tf(/)| 2 . 

(2) For the very important special case where H(f) is a rational 
function, i.e., H(f) = P(i2irf)/Q(i2irf), and P, Q are polynomials 
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with t- v ie degree of P < degree of Q, then conditions (i) and (ii) are 
sa tisfed. Since H(f) has only a finite number of zeros, condition (Hi) 
is pso satisfied. 

(3) Suppose that H(f) has no more than a countable number of 
zeros, but that S(f) = for some / £ [0, 1/7 1 ]. It is easy to see that 
some arbitrarily small change in T will cause S(f) to be strictly 
positive for all / 6E [0, 1/7']. Thus, condition (Hi) is not especially 
restrictive. 

We now state our main result, the proof of which is in Section III. 

Theorem 1 : Let h satisfy conditions (i), (ii), and (Hi) above. Then, for 
all e > 0, there exists a t = ro(e) sufficiently large so that, for all 

T > TO, 

PcN(D hr ) ^+ N V>r,c 2 (l + ■)*], 
where 

U(0, \t\ ^ T, 
MO - I (19) 

10, 1*1 > T, 

is the truncated version of h(t). The quantity t does not depend on N. 
Since h T has finite support, we conclude from Theorem 1 and (13) 
that, for all e > and r sufficiently large, 



dirh [1 + °< (1)] 



(20a) 



Kt{h, <t 2 ) ^ P eN (D hr ) 
^ exp 

[where o 2 (l) is independent of N"] so that 

E N (h) = - lim inf <r* log P>(h, J) ^ * ( * r) v, - (20b) 

We show in Appendix B that 

d(h r ) -> d(h), asr->«>, (21) 

so that letting e — » and t — » oo in (20b) yields 

A'.vW ^ ^- (22) 

We state this as 

Corollary 2: Let h satisfy conditions (i) to (Hi) above. Then, as a 2 — > 0, 



P* eN (h, <r 2 ) ^ exp 



-^D + o.(i)]j, 



where o 3 (l) is independent of N. 

We conclude this section with a remark concerning the relationship 
of the bound of Forney et al. (11) with the result of Corollary 2. We 
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can rewrite (11) as 

P*e*(h,o*) = Ki(*o,a 2 )e- rf2( * ,/2ff2 , 
and the bound of Corollary 2 as 

PUh, a 2 ) ^ K 2 (\\h\\i, '»> o>)e- di ^>*°\ (24) 

Here, we made explicit the dependence of K x on the support interval 
to of h(t) [see (9)], and the dependence of K 2 on \\h\\ h the £ x norm 
of h, and on m = mm S(f). Both Ki and K 2 increase in 1/a 2 slower 
than e dHh)li " 2 . But Ki(U, a 2 ) ->°o as < ->°o, and K 2 (||/i||i, m, a 2 ) -+», 
as ii^u x — ,. oo or as m _>o. Thus, although it might seem reasonable 
to assume that all h(t) satisfy (9) for some to, the bound of (23) depends 
on that h and becomes meaningless as /o-** 3 - Similarly, although 
it might be reasonable to assume for any h(t) that ||/i||i < °° and 
m = mm S(f) > 0, the bound of (24) depends on these quantities 
and also becomes meaningless as ||/i||i — >«= or m -> 0. Therefore, both 
bounds have their limitations; the new one, however, is considerably 
less limited. 

III. PROOF OF THEOREM 1 

Let h satisfy (i) to (in). Let h T (t) be as denned in (19), and let 
hr(t) = h(t) - hr(t), i.e., 

I (t) = °» I'l ^ T < (25) 

Then, if the data sequence is uG a*, the received sequence is y 
= h*u + z = /t T *u + 2, where 

z = z + hr*u. (26) 

Following the same steps as in (12), we obtain 

PMDk r ) 4? ^ 2_( ' V_1) 
W t b€CW 

Pr U {<S,Ar*(v-u)> fe*||Ar*(v-u)in, (27) 

where z is given in (26). 

We will show that, for arbitrary t > 0, there exists a t = r (e, /i) 
(t independent of N), such that for t ^ t , the event 

i(z,h T *(v-Xl)) ^ J||M(V-U)|| 2 } (28) 

c{ <z, / iT *(v - u)) ^ !L±^||M(v - u)|| 2 ), 

for all u G flt + , v G &%• Substituting (28) into (27) yields, on com- 
parison with (12), 

P eN (D hr ) £+Nlh r ,*{l + «) a ], 
which is Theorem 1. It remains to establish (28). 
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Relation (28) will follow immediately when we show the existence 
of a T (e, /*) such that, for t ^ to and all u, v, 



|</7,*u, /> T *(v - u))| g ^||/(r*(v - u)|| 2 . 
If (29) holds, the event in the left member of (28) 

i<Z,/<r*(v-u)> ^ J||Ar*(V- 1l)|| 2 } 

= \(Z, /, T *(V-U)) ^ ^||/ iT *(v — ll)|| 2 — (hr*U, h T *(V -u)) 



(29) 



C <2,M(V-U)> ^ 



1 + «. 



A r *(v — u)|| 2 



, (30) 



which is the right member of (28). Thus, it remains to establish (29). 
Let w = (it'o, • • ■ , iv jv-i) = v — u. The entries of w are 0, ±2. 
Also set q = h T *u, and r = // T *w. Then 

|v*,*u,M(v-u))| = |<g,r>| ^ £ \q(t)\\r{t)\dt 

g[ sup | q (01] I" \r(t)\dl. (31) 

-«o<|<« 7-00 

Consider 
f X \r(t)\dl = /"" E M« - fcT)i0* eft 

J-x, J-v> I *=0 

^ E W r |M* - kT)\dt ^ || A ||i £ |w»|, (32) 



where \\h\\i = f\h{t)\dt <qo, by condition (i). 

We obtain an upper bound on E |tp t | as follows. Since «;* = 0, ±2, 
we have £ \w k \ = ^ E «i Now, let H T (f) = /_% h T (t)e i2 "'dt be 
the Fourier transform of //,(/), and let 

M/) = B fJffr(/+!)| 2 , 0^/^^ (33) 

be the corresponding folded spectrum. Then, from Parseval's theorem, 

,i2*kT/\2df 



r|| 2 = ||Mw||' - r | #r(/) I 2 IE *<•',< 

7-00 fc 

= f%r(/)l2>*e a ** r W 

Jo k 

^ [ inf S r (/)] f^ IE w*e a ** r/ |W 

o</'<i/7' yo 



= [inf £,(/)] E u* = Dnf 5 r (/)] E |«'*| - ? - (34) 

/ A: 7 



Therefore, 



E |tr*| S*||r||*[ inf £,(/)]-'. 
a- og/gi/r 



(35) 
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Combining (31), (32), and (35), we have 

|< fff r)| £[ sup Jq(t)\T o nrf flr(i)]- 1 ilWIilkll 1 . (36) 

Now, we show in Appendix A that 

liminf [ inf 5 T (/)] ^ w > 0, (37) 

where m = min*S(/) > [see condition (m)]. Further, using condi- 
tion (i) [particularly the monotonicity of g (0 ], we nave 

I a CO I = I LM< - Ar)u*| ^ E |X r (* - *D1 

< £ |l,(i - kT)\ = E |M< -MO I 

^ E J7o«-fcD^ f CflroCr+iD +?o(-r-.;T)] 

fc:|t-3Er| 2t j'-o 

z\i[ m g*{t)dt+\,r go(t)dt-*0, asr-^co. (38) 

i y T -r i J-t+t 

Combining (36), (37), and (38), we obtain 

' (g, r) 1 \(h r *u, My - u))l ^ 
||r|| 2 p T *(v - u|| 2 

as T _>oo. This is equivalent to (29), so that the proof of Theorem 1 
is complete. 
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APPENDIX A 

The Folded Spectrum S(f) 

We first show that S(f) as given in (17) is always finite, i.e., the 
series in (17) converges for all / G [0, 1/T]. From condition (it), 
using the monotonicity of G\, 

*(*„,/) = E k(/+5)|V £ °*(/+j) 

I n| gno | \ ■* / I |n| Sno \ 1 / 

= E Gi(/ + |)+ E G 1 (/+^) l 

+ r f Gi(aj)dx->0, as n -► «>, (39) 

Jn,-1IT 

so that the series in (17) converges. 
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To establish the continuity of S(f), write 

s(f)= E |^f/ + 5)r + «(wo./), OS/Sl/T. (40) 

|n| S"o |\ i / | 



For arbitrary S.Og/H/T, 



W)-W+«)l ^ 



^+f)r-K' +,+ *)n 

+ l€(no, /)l + I *(«o, /+*)!• (41) 

Now since /t(/) E«£i( — °°, co )> #(/) is continuous. To make the 
right member of ineq. (41) ^e, first let n be sufficiently large so that 
the last two terms of the right member of ineq. (41) ^e/2; then 
choose | 5 1 sufficiently small so that the first term of the right member 
of inequality (41) ^e/2. This establishes the continuity of S(f). 

Wenext verify (37), which concerns S T (f). Since h T is in£i( — «, »), 
H T (f) exists for all /G (- °°, «). Thus, S T (f) as defined in (33) is 
meaningful, though perhaps infinite on a set of measure zero. With 
£(n /) as in (39), write 

I / « \ I 2 " I / « \ I 2 

SAf)-S(f)= L \HJf + %)\ - E \H(f+%)\ 

^ e rk'(/+5)r-k(/+^)ri- €(no ' /) - (42) 

Now let e > be arbitrary. From (39) we can choose no sufficiently 
large such that £(n ,/) g e/2, for/ £ [0, 1/T]. With n Q so chosen, 



(43) 



^ (/) - w) ^ n sjK /+ i)i 2 -K /+ ^)r]-2 e ' 

Now let HrU) be the Fourier transform of h T . Then 

|#(/)| = |^(/)+^r(/)| ^ |#r(/)| + \H T (f)\. 

Therefore 

|#(/)| 2 ^ |// T (/)| 2 + 2|tf T (/)||#(/)| + |# T (/)| 2 
^ |# T (/)| 2 + 2||MiPr||i + \\h T \\l 

where | ||i denotes £i norm. Since ||/z r ||i— >0, as r — > «, if r is 
sufficiently large, then 



|#x(/)| - \H(f)\ ^ - - 



1 



2 (2w + 1) 



, /€(-«, co). (44) 



Inequalities (43) and (44) imply that, for all e > 0, there exists a 
To(e) such that for all r ^ ro(e), 



S T (/) ^S(f)- e, 0^/^y, 



(45) 
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Thus, for r ^ T-o(e), 

inf SAf)^ inf S(f) - « = m - e. (46) 

Letting t -» <» and e -> in (46) yields (37). 

APPENDIX B 

Convergence of the Minimum Distance 

In this appendix, we shall verify (21), i.e., 

d(h T )^d{h), as r -> ». (47) 

From the definition of d(/i T ) (10), for arbitrary e > 0, we are assured 
of the existence of a w = u — v such that u, v E Gat, and 

||/tr*w|| ^ d(/;, T ) + e. (48) 

Repeating the steps in (34), we obtain 

\\hr*w\\ 2 = f llT Sr(f)\Y, w k e*"< T '\ 2 df ^ 2[inf£ T (/)] £* |i0»|. (49) 

From (37) we can choose r sufficiently large so that 



inf»S r (/) ^ m. (50) 

/ 



Hence, for such a choice of t, 



N-l 

E |wfc| = .) 



TOj + 6? (gl) 



Now 

d(h) ^ P*w|| = ||Ar*w - ^ T *w|| ^ pr*w|| + ||^t*w|| 

^ d(fc T ) + c + ||A r *w||. (52) 

Since 

(A r *w)(0 = EM* - fcTO-w*, 

it 

we have, with r large enough to satisfy (51), 

P T *W|| ^ E I «>* | Px|| = ||Xr|| E |»*| ^ fe"" Cd(*r) + «] 2 - (53) 

Combining (52) and (53) yields for r sufficiently large (and e > 
arbitrary) 

d(h) ^ d(h T ) + 6 + fell [d(A T ) + e] 2 . (54) 
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Letting t — > *> and e — » yields 

d{h) ^ liminf d(h T ). (55) 

T-»cc 

The identical argument with h and h r reversed yields for all t > 0, 
e > 0, 

d(h T ) ^ d{h) + e + W [d(fc) + tf, 

so that (letting r — ► °c , « — > 0) 

lim sup d(/i T ) ^ d(h). (56) 

r-»oo 

Inequalities (55) and (56) yield (47) or (21), completing the proof. 
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